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Detail Action 



1. 



This office action is in response to the application filed on 7/1 1/2003. 



2. 



Claims 1-49 are pending. 



Restriction 



3. Restriction to one of the following inventions is required under 35 U.S.C. 121 : 

I. Claims 1-22, 34-41, and 48, drawn to speech controlled system, classified 
in class 704, subclass 275. 

II. Claims 23-33, 42-47, and 49, drawn to software program development 
tool, classified in class 717, subclass 100. 

The inventions are distinct, each from the other because of the following reasons: 

Inventions I and II are related as process of making and product made. The 
inventions are distinct if either or both of the following can be shown: (1) that the 
process as claimed can be used to make another and materially different product or (2) 
that the product as claimed can be made by another and materially different process 
(MPEP § 806.05(f)). In the instant case, group I is related to a multimodal software 
application and group II is related to developing multimodal software applications. 

Because these inventions are independent or distinct for the reasons given 
above and there would be a serious burden on the examiner if restriction is not required 
because the inventions have acquired a separate status in the art in view of their 
different classification, restriction for examination purposes as indicated is proper. 
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Based on phone message from Kurt Summe (Reg. No. 36,023) on December 
29, 2006, a provisional election was made without traverse to prosecute the invention of 
group I, claims 1-22, 34-41, and 48 . Affirmation of this election must be made by 
applicant in replying to this Office action. Claims 23-33, 42-47, and 49 withdrawn from 
further consideration by the examiner, 37 CFR 1.142(b), as being drawn to a non- 
elected invention. 

Applicant is reminded that upon the cancellation of claims to a non-elected 
invention, the inventorship must be amended in compliance with 37 CFR 1 .48(b) if one 
or more of the currently named inventors is no longer an inventor of at least one claim 
remaining in the application. Any amendment of inventorship must be accompanied by 
a request under 37 CFR 1.48(b) and by the fee required under 37 CFR 1.1 7(i). 

Specification 

4. The disclosure is objected to because of the following informalities: page 1 of the 
specification contains a blank application serial number. 

Appropriate correction is required. 

Claim Objections 

5. Claims 3 and 17 are objected to because of the following informalities: It appears 
that the claim body has misplaced speech recognizer and speech synthesizer. The 
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claim body should be "wherein the first set of peripheral devices includes a speech 
recognizer and the second set of peripheral devices includes a speech synthesizer". 
Appropriate correction is required. 



Claim Rejections - 35 USC § 101 

35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

6. Claim 48 is rejected under 35 U.S.C. 101 because the claimed invention is 

directed to non-statutory subject matter. 

Current Patent Office practice does not consider signal as a patentable subject 
matter because signal is considered as a form of energy. 

Claim 48 recites the limitation of a computer-readable medium. According to the 
specification, page 11, line 5-10, "...Examples of signal bearing media include but are 
not limited to... among others, and transmission type media such as digital and analog 
communication links." Such communication links can be signals transmitted from one 
point to another point. It can be signal, therefore a non-patentable subject matter. 



Claim Rejections - 35 USC §112 

The following is a quotation of the second paragraph of 35 U.S.C. 112: 
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The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

7. Claims 1-15, and 18 are rejected under 35 U.S.C. 112, second paragraph, as 
being indefinite for failing to particularly point out and distinctly claim the subject matter 
which applicant regards as the invention. 

Claims 1 and 18 recite the limitation of "the associated peripheral device" in "to 
the associated peripheral device" and "from the associated peripheral device". There is 
insufficient antecedent basis for this limitation in the claim. Claims 2-15 depend on 
claim 1 and suffer the same deficiency as claim 1 . For purpose of art rejection, the 
examiner will assume "the associated peripheral device" as "an associated peripheral 
device". 

Further claim 2, the phrase "can be" renders the claim indefinite because it is 
unclear if there are other possibilities for a peripheral device that are not listed in the 
claim. 

Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 
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(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

8. Claims 1-4, 10, and 16-19 are rejected under 35 U.S.C. 102(b) as being 
anticipated by Takayama et al. (US Patent No. 6,330,539). 

As per claim 1 , 
Takayama et al. disclose 

A system for executing a multimodal software application, comprising: 

- the multimodal software application, wherein said multimodal software 
application is configured to receive first data input from a first set of 
peripheral devices and output second data to a second set of peripheral 
devices (c4 : 24-35, "...converting input speech. ..outputting 
an output semantic representation../') ; 

- a dialog engine in communication with the multimodal software 
application, wherein said dialog engine is configured to execute a 
workflow description related to the multimodal software application and 
provide the first data to the multimodal software application; and a 
respective interface component associated with each peripheral device 
within said first and second sets; wherein each interface component is 
configured to provide the second data, if any, to an associated peripheral 
device and receive the first data, if any, from an associated peripheral 
device (c4 : 36-44 , "...an innate operation unit for 
receiving an innate operation command from the dialog 



Application/Control Number: 10/617,422 Page 7 

Art Unit: 2191 

management unit../ 7 ; c4 : 53-64, "...comprise dialog rules 
storage unit../ 7 The examiner asserts that the dialog 
rules is a workflow description; c7 : 45-57 & Fig. 4; 
cl: 5-9, "...a user interface adaptable. ..to a computer 
or an electrical device../ 7 ) . 

As per claim 2, 

the rejection of claim 1 is incorporated; 
further Takayama et al. disclose 

- wherein a peripheral device cab be a member of both the first and second 

sets (c9:l-2, "...it is possible to integrate several of 
the same hardware parts into a single element... 77 ) . 

As per claim 3, 

the rejection of claim 1 is incorporated; 
further Takayama et al. disclose 

- wherein the first set of peripheral devices includes a speech recognizer 

and the second set of peripheral devices includes a speech synthesizer 

(c8:41-43, "..speech recognition apparatus...a speech 
syntheses apparatus../ 7 ) . 

As per claim 4, 

the rejection of claim 1 is incorporated; 
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further Takayama et al. disclose 

- wherein the multimodal software application further comprises a graphical 
user interface including a screen (c4 : 6-13, "...it can be 
included as a part of the above-described multimodal 
interface../ 7 ; cl: 25-30, "...GUI"; this shows that GUI is 
widely used in multi-modal application) . 

As per claim 10, 

the rejection of claim 1 is incorporated; 
further Takayama et aL disclose 

- wherein the dialog engine is further configured to redirect the first data to 
a third set of peripheral devices comprising selected members from the 
first and second set (c3: 13-17 and Fig. 3). 

As per claim 16, 
Takayama et al. disclose 

A system for executing a multimodal software application comprising: 

- a dialog engine in communication with a) the multimodal software 
application, b) a first set of peripheral devices for receiving first data, and 
c) a second set of peripheral devices for outputting second data; and 
said dialog engine configured to execute a workflow description related to 
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the multimodal software application, wherein executing the workflow 
description includes generating the second data from the workflow 
description and providing the first data to the multimodal software 
application (c4: 36-44, "...an innate operation unit for 
receiving an innate operation command from the dialog 
management unit../'; c4 : 53-64, "...comprise dialog rules 
storage unit../ 7 The examiner asserts that the dialog 
rules is a workflow description; c7 : 45-57 & Fig. 4). 

As per claim 17, 

- see reason for rejection of claim 3. 

As per claim 18, 

the rejection of claim 16 is incorporated; 
Takayama et al. disclose 

- wherein the dialog engine is configured to communicate with each of the 
peripheral devices in the first and second sets via a respective interface 
component associated with each peripheral device within said first and 
second set; wherein the interface component is configured to provide 
the second data, if any, to an associated peripheral device and receive 
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the first data, if any, from an associated peripheral device (c4 : 36-4 4 , 
"...an innate operation unit for receiving an innate 
operation command from the dialog management unit..."; 
c4 : 53-64, "...comprise dialog rules storage unit..." 
The examiner asserts that the dialog rules is a 
workflow description; c7 : 45-57 & Fig. 4; cl: 5-9, 
"...a user interface adaptable. ..to a computer or an 
electrical device... ") 

As per claim 19, 

- See reason for rejection of claim 4. 



Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

9. Claims 5-9, 11-15, 20-22, 34-41, and 48 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Takayama et al. (US Patent No. 6,330,539) in view of 
Peterson et al. (US Patent No. 7,039,166). 
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As per claim 5, 

the rejection of claim 4 is incorporated; 
Takayama et al. do not specifically disclose 

- wherein the workflow description comprises a set of workflow objects, 

wherein each workflow object is associated with a respective visual 
control within said screen. 
However, Peterson et al. disclose 

- wherein the workflow description comprises a set of workflow objects, 

wherein each workflow object is associated with a respective visual 
control within said screen (c2:45-54, "...visually 
representing../ 7 ; Fig. 14 and c33: 42-66, "...The nodes of 
the tree corresponds to IVR states or prompts, arcs 
correspond to state transitions../') . 
Therefore, it would have been obvious to a person of ordinary skill in the art at the time 
the invention was made to incorporate the teachings of Peterson et al. into the 
teachings of Takayama et al. to include a set of workflow objects, wherein each 
workflow object is associated with a respective visual control within said screen. The 
modification would be obvious to one of ordinary skill in the art to want to enable 
assessment of the system as suggested by Peterson et al. (c2 : 47-49, "...enable 
assessment../') . 
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As per claim 6, 

the rejection of claim 5 is incorporated; 

Takayama et al. do not specifically disclose the claim limitation, 

However, Peterson et al. disclose 

- wherein each workflow object further comprises: a prompt related to the 
associated visual control; and a link to another workflow object (c3 : 
47-67). 

As per claim 7, 

the rejection of claim 6 is incorporated; 

Takayama et al. do not specifically disclose the claim limitation, 

However, Peterson et al. disclose 

- wherein each workflow object further comprises: a plurality of expected 
input values; and a help message (c4 : 60-64, "...trigger by data 
inputted by a user or by internal processing../'. To 
enable such triggering, data inputted must match 
certain criteria set by the system, therefore the 
expected input values; c3 : 51, x \..a first prompt 
generated upon entry../' The examiner asserts this 
prompt is a help message.). 
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As per claim 8, 

the rejection of claim 6 is incorporated; 
Takayama et al. disclose 

- wherein each workflow object further comprises: a first identification of 
members of the first set from which first data can be received; and a 
second indication of members of the second set to which second data can 
besent(c5: 9-16, "... identifying an origin of the input 
speech../'; c5 : 17-26, "...data specifying the dialog 
target../') . 

As per claim 9, 

the rejection of claim 6 is incorporated; 

Takayama et al. do not specifically disclose the claim limitation, 

However, Peterson et al. disclose 

- wherein the prompt is the second data (c3: 52-53, "...subsequent 
prompts triggered by data inputted by a user...") . 

As per claim 1 1 , 

the rejection of claim 6 is incorporated; 
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Takayama et al. do not specifically disclose the claim limitation, 
However, Peterson et al. disclose 

- wherein each workflow object further comprises: a plurality of links, each 
link being to a different respective workflow object and each of the 
plurality of links having an activation criterion (c4:53-63). 

As per claim 12, 

the rejection of claim 11 is incorporated; 

Takayama et al. do not specifically disclose the claim limitation, 

However, Peterson et al. disclose 

- wherein the activation criterion relates to a value of the first data(c4:53- 
63). 

As per claim 13, 

the rejection of claim 6 is incorporated; 
Takayama et al. disclose 

- instruct each interface component associated with a respective member 
of the first set to wait for the first data (c9: 54-64, "...an out-wait 
signal../ 7 ; see FIG . 11, step S12) ; 

Takayama et al. do not specifically disclose 
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- wherein the dialog engine is further configured to: execute a particular 
workflow object by outputting the prompt as the second data; based on 
the first data determine whether to follow the link; and execute said 
another workflow object 

However, Peterson et al. disclose 

- wherein the dialog engine is further configured to: execute a particular 
workflow object by outputting the prompt as the second data; based on 
the first data determine whether to follow the link; and execute said 
another workflow object (c4: 53-64, "). 

As per claim 14, 

the rejection of claim 6 is incorporated; 
Takayama et al. disclose 

- wherein the dialog engine is further configured to: receive the first data; 
and forward the first data to each interface component associated with a 
respective member of the second set (c5: 8-26). 

As per claim 15, 

the rejection of claim 4 is incorporated; 

Takayama et al. do not specifically disclose the claim limitation, 
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However, Peterson et al. disclose 

- wherein the execution of the workflow description is synchronized to a 
display, by the dialog engine, of the graphical user interface (c2 : 4 5- 
46, "... visually representing the behavior of a user../') . 

Therefore, it would have been obvious to a person of ordinary skill in the art at the time 
the invention was made to incorporate the teachings of Peterson et al. into the 
teachings of Takayama et al. to include the execution of the workflow description is 
synchronized to a display, by the dialog engine, of the graphical user interface. The 
modification would be obvious to one of ordinary skill in the art to want to enable 
assessment of the system as suggested by Peterson et al. (c2 : 47-4 9, "...enable 
assessment../') . 

As per claim 20, 

- See reason for rejection of claim 5. 

As per claim 21, 

- See reason for rejection of claim 6. 

As per claim 22, 

the rejection of claim 21 is incorporated; 
Takayama et al. disclose 
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- wherein the dialog engine further comprises: an execution unit configured 
to execute a particular workflow object (c4: 53-64); 

Takayama et al. do not specifically disclose the rest of the limitation. 
However, Peterson et al. disclose 

- a prompt generator configured to output the prompt as the second data 
(c3: 52-53, "... subsequent prompts triggered by data 
inputted by a user../'); 

- a data tester configured to determine if the first data received in response 
to the second data satisfies a set of criteria associated with the link 

(c3: 52-53, "...subsequent prompts triggered by data 
inputted by a user../'); 

- and an object loader configured to load the another workflow object in the 
execution unit when instructed by said data tester (c3: 4 6-64, 

. "...cause a branching in a call flow. ..a start state. ..other 
states of the finite-sate machine represent subsequent 
prompts at which a branching occurs../') . 

As per claim 34, 
Takayama et al. disclose 

A method for executing a multimodal software application having a graphical user 
interface with a screen, the method comprising 
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- the steps of: receiving a workflow description corresponding to the screen 

(c4: 54-67, "...the dialog management unit has functions 
for making deletions, changes, and additions with 
respect to the dialog rules../';) 
Takayama et al. do not specifically disclose 

- and executing the workflow description in synchronization with the 
graphical user interface. 

However, Peterson et al. disclose 

- and executing the workflow description in synchronization with the 
graphical user interface (c2: 4 5-54, "...visually 

representing../' ;Fig . 14 and c33: 42-66, "...The nodes of. 
• the tree corresponds to IVR states or prompts, arcs 

correspond to state transitions../') . 
Therefore, it would have been obvious to a person of ordinary skill in the art at the time 
the invention was made to incorporate the teachings of Peterson et al. into the 
teachings of Takayama et al. to include and executing the workflow description in 
synchronization with the graphical user interface. The modification would be obvious to 
one of ordinary skill in the art to want to enable assessment of the system as suggested 
by Peterson et al. (c2: 47-49, "...enable assessment../'). 
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As per claim 35, 

the rejection of claim 34 is incorporated; 
further Perterson et al. disclose 

- wherein the step of executing includes the steps of: identifying a workflow 
object associated with a visual control on the screen; executing the 
workflow object; and identifying another workflow object linked to the 
workflow object (c3:46-64). 

As per claim 36, 

the rejection of claim 35 is incorporated; 
Takayama et al. disclose 

- instructing a second set of peripheral devices to wait for a response(c9 : 
54-64, "...an out-wait signal../'; see FIG. 11, step S12); 

- and forwarding the response to the multimodal software application 
(c4:25-44). 

further Peterson et al. disclose 

- extracting a prompt from the workflow object; sending the prompt to a first 
set of peripheral devices; (c3:46-64). 

As per claim 37, 

the rejection of claim 36 is incorporated; 
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Takayama et al. disclose 

- further comprising the step of: forwarding the response to one or more of 
the peripheral devices of the first and second sets (Fig. 3). 

As per claim 38, 

the rejection of claim 36 is incorporated; 
further Peterson et al. disclose 

- further comprising the steps of: extracting a default help prompt from the 
workflow object; and sending the default help prompt to one or more of 
the peripheral devices of the first and second sets (c3: 46-64). 

As per claim 39, 

the rejection of claim 34 is incorporated; 
further Peterson et al. disclose 

- wherein the step of executing includes the steps of: outputting audio 
prompts corresponding to the screen; and receiving input via speech 
recognition system in response to the audio prompts (c3:46-64). 

As per claim 40, 

the rejection of claim 36 is incorporated; 
Takayama et al. disclose 
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- further comprising the steps of: extracting information regarding 
peripheral device membership in the first and second sets from the 
workflow Object (c5: 9-16, "...identifying an origin of the 
input speech../'; c5:. 17-26, "...data specifying the 
dialog target../') 

As per claim 41, 

the rejection of claim 36 is incorporated; 
further Peterson et al. disclose 

- wherein the response also includes data related to the another workflow 
object(c3:46-64). 

As per claim 48, 

- it is the computer-readable medium claim corresponding to method claim 
34 and is rejected for the same reason set forth in connection of the 
rejection of claim 34 above. 



Conclusion 



The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 



Application/Control Number: 10/617,422 



Page 22 



Art Unit: 2191 

It is noted that any citation [[s]] to specific, pages, columns, lines, or figures in the prior 
art references and any interpretation of the references should not be considered to be limiting in 
any way. A reference is relevant for all it contains and may be relied upon for all that it would 
have reasonably suggested to one having ordinary skill in the art. [[See, MPEP 2123]] 

Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Philip Wang whose telephone number is 571-272-5934. The examiner 
can normally be reached on Mon - Fri 8:00AM - 4:00PM. Any inquiry of general nature or 
relating to the status of this application should be directed to the TC21 00 Group receptionist: 
571-272-2100. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Wei Zhen can be reached on 571-272-3708. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
Information regarding the status of an application may be obtained from the Patent Application 
Information Retrieval (PAIR) system. Status information for published applications may be 
obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private 
PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 




